Lightweight Asynchronous Snapshots for Distributed Dataflows

Distributed stateful stream processing enables the deployment and execution of large scale continuous computations in the cloud, targeting both low latency and high throughput. One of the most fundamental challenges of this paradigm is providing processing guarantees under potential failures. Existing approaches rely on periodic global state snapshots that can be used for failure recovery. Those approaches suffer from two main drawbacks. First, they often stall the overall computation which impacts ingestion. Second, they eagerly persist all records in transit along with the operation states which results in larger snapshots than required. In this work we propose Asynchronous Barrier Snapshotting (ABS), a lightweight algorithm suited for modern dataflow execution engines that minimises space requirements. ABS persists only operator states on acyclic execution topologies while keeping a minimal record log on cyclic dataflows. We implemented ABS on Apache Flink, a distributed analytics engine that supports stateful stream processing. Our evaluation shows that our algorithm does not have a heavy impact on the execution, maintaining linear scalability and performing well with frequent snapshots.Comment: 8 pages, 7 figure

A Survey on the Evolution of Stream Processing Systems

Stream processing has been an active research field for more than 20 years, but it is now witnessing its prime time due to recent successful efforts by the research community and numerous worldwide open-source communities. This survey provides a comprehensive overview of fundamental aspects of stream processing systems and their evolution in the functional areas of out-of-order data management, state management, fault tolerance, high availability, load management, elasticity, and reconfiguration. We review noteworthy past research findings, outline the similarities and differences between early ('00-'10) and modern ('11-'18) streaming systems, and discuss recent trends and open problems.Comment: 34 pages, 15 figures, 5 table

Haren: A Framework for Ad-Hoc Thread Scheduling Policies for Data Streaming Applications

In modern Stream Processing Engines (SPEs), numerous diverse applications, which can differ in aspects such as cost, criticality or latency sensitivity, can co-exist in the same computing node. When these differences need to be considered to control the performance of each application, custom scheduling of operators to threads is of key importance (e.g., when a smart vehicle needs to ensure that safety-critical applications always have access to computational power, while other applications are given lower, variable priorities).Many solutions have been proposed regarding schedulers that allocate threads to operators to optimize specific metrics (e.g., latency) but there is still lack of a tool that allows arbitrarily complex scheduling strategies to be seamlessly plugged on top of an SPE. We propose Haren to fill this gap. More specifically, we (1) formalize the thread scheduling problem in stream processing in a general way, allowing to define ad-hoc scheduling policies, (2) identify the bottlenecks and the opportunities of scheduling in stream processing, (3) distill a compact interface to connect Haren with SPEs, enabling rapid testing of various scheduling policies, (4) illustrate the usability of the framework by integrating it into an actual SPE and (5) provide a thorough evaluation. As we show, Haren makes it is possible to adapt the use of computational resources over time to meet the goals of a variety of scheduling policies

GeneaLog: Fine-Grained Data Streaming Provenance at the Edge

Fine-grained data provenance in data streaming allows linking each result tuple back to the source data that contributed to it, something beneficial for many applications (e.g., to find the conditions triggering a security- or safety-related alert). Further, when data transmission or storage has to be minimized, as in edge computing and cyber-physical systems, it can help in identifying the source data to be prioritized.The memory and processing costs of fine-grained data provenance, possibly afforded by high-end servers, can be prohibitive for the resource-constrained devices deployed in edge computing and cyber-physical systems. Motivated by this challenge, we present GeneaLog, a novel fine-grained data provenance technique for data streaming applications. Leveraging the logical dependencies of the data, GeneaLog takes advantage of cross-layer properties of the software stack and incurs a minimal, constant size per-tuple overhead. Furthermore, it allows for a modular and efficient algorithmic implementation using only standard data streaming operators. This is particularly useful for distributed streaming applications since the provenance processing can be executed at separate nodes, orthogonal to the data processing. We evaluate an implementation of GeneaLog using vehicular and smart grid applications, confirming it efficiently captures fine-grained provenance data with minimal overhead

Gravitational Waves From Known Pulsars: Results From The Initial Detector Era

We present the results of searches for gravitational waves from a large selection of pulsars using data from the most recent science runs (S6, VSR2 and VSR4) of the initial generation of interferometric gravitational wave detectors LIGO (Laser Interferometric Gravitational-wave Observatory) and Virgo. We do not see evidence for gravitational wave emission from any of the targeted sources but produce upper limits on the emission amplitude. We highlight the results from seven young pulsars with large spin-down luminosities. We reach within a factor of five of the canonical spin-down limit for all seven of these, whilst for the Crab and Vela pulsars we further surpass their spin-down limits. We present new or updated limits for 172 other pulsars (including both young and millisecond pulsars). Now that the detectors are undergoing major upgrades, and, for completeness, we bring together all of the most up-to-date results from all pulsars searched for during the operations of the first-generation LIGO, Virgo and GEO600 detectors. This gives a total of 195 pulsars including the most recent results described in this paper.United States National Science FoundationScience and Technology Facilities Council of the United KingdomMax-Planck-SocietyState of Niedersachsen/GermanyAustralian Research CouncilInternational Science Linkages program of the Commonwealth of AustraliaCouncil of Scientific and Industrial Research of IndiaIstituto Nazionale di Fisica Nucleare of ItalySpanish Ministerio de Economia y CompetitividadConselleria d'Economia Hisenda i Innovacio of the Govern de les Illes BalearsNetherlands Organisation for Scientific ResearchPolish Ministry of Science and Higher EducationFOCUS Programme of Foundation for Polish ScienceRoyal SocietyScottish Funding CouncilScottish Universities Physics AllianceNational Aeronautics and Space AdministrationOTKA of HungaryLyon Institute of Origins (LIO)National Research Foundation of KoreaIndustry CanadaProvince of Ontario through the Ministry of Economic Development and InnovationNational Science and Engineering Research Council CanadaCarnegie TrustLeverhulme TrustDavid and Lucile Packard FoundationResearch CorporationAlfred P. Sloan FoundationAstronom

Maternal hormonal milieu influence on fetal brain development

An adverse maternal hormonal environment during pregnancy can be associated with abnormal brain growth. Subtle changes in fetal brain development have been observed even for maternal hormone levels within the currently accepted physiologic ranges. In this review, we provide an update of the research data on maternal hormonal impact on fetal neurodevelopment, giving particular emphasis to thyroid hormones and glucocorticoids. Thyroid hormones are required for normal brain development. Despite serum TSH appearing to be the most accurate indicator of thyroid function in pregnancy, maternal serum free T4 levels in the first trimester of pregnancy are the major determinant of postnatal psychomotor development. Even a transient period of maternal hypothyroxinemia at the beginning of neurogenesis can confer a higher risk of expressive language and nonverbal cognitive delays in offspring. Nevertheless, most recent clinical guidelines advocate for targeted high-risk case finding during first trimester of pregnancy despite universal thyroid function screening. Corticosteroids are determinant in suppressing cell proliferation and stimulating terminal differentiation, a fundamental switch for the maturation of fetal organs. Not surprisingly, intrauterine exposure to stress or high levels of glucocorticoids, endogenous or synthetic, has a molecular and structural impact on brain development and appears to impair cognition and increase anxiety and reactivity to stress. Limbic regions, such as hippocampus and amygdala, are particularly sensitive. Repeated doses of prenatal corticosteroids seem to have short-term benefits of less respiratory distress and fewer serious health problems in offspring. Nevertheless, neurodevelopmental growth in later childhood and adulthood needs further clarification. Future studies should address the relevance of monitoring the level of thyroid hormones and corticosteroids during pregnancy in the risk stratification for impaired postnatal neurodevelopment.This work was supported by the grant "Doutoramento em Medicina Jose de Mello Saude 2014" by Jose de Mello Saude to AM

Measurement of ϒ production in pp collisions at √s = 2.76 TeV

The production of ϒ(1S), ϒ(2S) and ϒ(3S) mesons decaying into the dimuon final state is studied with the LHCb detector using a data sample corresponding to an integrated luminosity of 3.3 pb−1 collected in proton–proton collisions at a centre-of-mass energy of √s = 2.76 TeV. The differential production cross-sections times dimuon branching fractions are measured as functions of the ϒ transverse momentum and rapidity, over the ranges pT < 15 GeV/c and 2.0 < y < 4.5. The total cross-sections in this kinematic region, assuming unpolarised production, are measured to be σ (pp → ϒ(1S)X) × B ϒ(1S)→μ+μ− = 1.111 ± 0.043 ± 0.044 nb, σ (pp → ϒ(2S)X) × B ϒ(2S)→μ+μ− = 0.264 ± 0.023 ± 0.011 nb, σ (pp → ϒ(3S)X) × B ϒ(3S)→μ+μ− = 0.159 ± 0.020 ± 0.007 nb, where the first uncertainty is statistical and the second systematic

Search for dark matter produced in association with bottom or top quarks in √s = 13 TeV pp collisions with the ATLAS detector

A search for weakly interacting massive particle dark matter produced in association with bottom or top quarks is presented. Final states containing third-generation quarks and miss- ing transverse momentum are considered. The analysis uses 36.1 fb−1 of proton–proton collision data recorded by the ATLAS experiment at √s = 13 TeV in 2015 and 2016. No significant excess of events above the estimated backgrounds is observed. The results are in- terpreted in the framework of simplified models of spin-0 dark-matter mediators. For colour- neutral spin-0 mediators produced in association with top quarks and decaying into a pair of dark-matter particles, mediator masses below 50 GeV are excluded assuming a dark-matter candidate mass of 1 GeV and unitary couplings. For scalar and pseudoscalar mediators produced in association with bottom quarks, the search sets limits on the production cross- section of 300 times the predicted rate for mediators with masses between 10 and 50 GeV and assuming a dark-matter mass of 1 GeV and unitary coupling. Constraints on colour- charged scalar simplified models are also presented. Assuming a dark-matter particle mass of 35 GeV, mediator particles with mass below 1.1 TeV are excluded for couplings yielding a dark-matter relic density consistent with measurements